Skip to content

feat(compiler): Add grpc support for Swift - #3776

Merged
chaokunyang merged 51 commits into
apache:mainfrom
yash-agarwa-l:grpc-swift
Aug 24, 2026
Merged

feat(compiler): Add grpc support for Swift#3776
chaokunyang merged 51 commits into
apache:mainfrom
yash-agarwa-l:grpc-swift

Conversation

@yash-agarwa-l

Copy link
Copy Markdown
Contributor

Why?

Swift users can generate Fory model types today, but schemas that define services
do not produce Swift gRPC companions. This leaves Swift out of the existing --grpc
workflow used by the other supported service-generation targets.

What does this PR do?

  • Adds Swift gRPC companion generation for Fory compiler services, including service
    metadata descriptors, an EventLoopFuture provider, an async/await provider, an
    async client, and Fory-backed request/response stream adapters, targeting grpc-swift 1.x.
  • Serializes request and response bodies with Fory instead of protobuf through an
    internal GRPCPayload wrapper. Because the Swift Fory runtime is single-threaded,
    the wrapper builds one Fory per thread from the schema module's own configuration
    and registrations, so concurrent RPCs are race-free (verified under ThreadSanitizer).
  • Emits an async-only client; the EventLoopFuture client and interceptor hooks are
    omitted because their generated types would expose the internal wrapper.
  • Adds Swift preflight validation for generated output-path and top-level symbol
    collisions, and reserves inherited provider/client member names (handle,
    serviceName, channel, defaultCallOptions) so a clashing rpc fails codegen
    with a clear message.
  • Updates compiler and service-codegen tests to cover the four streaming shapes,
    identifier escaping, imported and nested service types, the default package, the
    protobuf and FlatBuffers frontends, collision handling, a SwiftPM build-and-run
    fixture, and a marshaller concurrency test under ThreadSanitizer (gated behind
    FORY_SWIFT_TSAN).
  • Wires Swift output into the cross-language gRPC generation helper.
  • Documents Swift gRPC support, dependencies, generated API shape, streaming, the
    shared-top-level-package limitation, troubleshooting, and compiler guide updates.

Draft: Java<->Swift cross-language interop tests (all four modes, all three IDL
frontends) are in progress and will be added before this PR is marked ready for review.

Related issues

#3266
#3370

AI Contribution Checklist

  • Substantial AI assistance was used in this PR: yes
  • If yes, I included a completed AI Contribution Checklist in this PR description and the required AI Usage Disclosure.
  • If yes, my PR description includes the required ai_review summary and screenshot evidence of the final clean AI review results from both fresh reviewers on the current PR diff or current HEAD after the latest code changes.

Does this PR introduce any user-facing change?

  • Does this PR introduce any public API change?
    • Adds generated Swift gRPC companion APIs when foryc --swift_out=... --grpc is used.
  • Does this PR introduce any binary protocol compatibility change?
    • The generated services use Fory-encoded gRPC message bodies, but this PR does not
      change the Fory binary protocol.

Benchmark

Not applicable.

Schemas with services now emit a <Service>Grpc.swift companion beside the
Swift model. Each service gets Fory-backed async and NIO providers plus an
async client; request and response bytes ride a private GRPCPayload wrapper
that serializes through the schema module's Fory instance.
Before writing Swift output, check that no two schemas or services claim the
same file path or top-level symbol. A service named after a generated type, or
a duplicate service, now fails fast with a clear message instead of emitting
Swift that will not compile.
Exercise the Swift companion across the four streaming shapes, keyword-escaped
methods, imported request and response types, the default package, both IDL
frontends, and the collision preflight, so the emitter and its symbol names stay
pinned.
Generate a two-package schema, then swift build and run a SwiftPM package on
grpc-swift and local Fory that hosts the generated provider and round-trips all
four streaming shapes across the import boundary. Skipped when swift is absent.
Wire Swift into the shared gRPC generation step so the interop schemas emit
Swift companions alongside the other targets.
Add a Swift gRPC guide covering dependencies, server and client usage, streaming,
and troubleshooting, link it from the Swift guide index, and note the Swift
companion in the compiler guide and agent rules.
Break the streaming handler closures across lines so generated companions stay
under the swiftlint line-length limit even with long package-qualified names.
Put handler braces on the declaration line, give each async parameter its own
aligned line, name the unwrapped stream value, and scope a type_name disable
around the package-prefixed symbols so swiftlint reports no violations.
Schemas that share a top-level package component make the model generator emit a
duplicate root enum, which the Swift compiler rejects in one module. Pin it with
a strict xfail fixture and a docs note pointing at disjoint packages.
The Swift Fory instance is single-threaded, but gRPC drives the marshaller from
many threads at once, so sharing one instance races. Build one Fory per thread
from the module config and registrations, and fire 200 parallel calls in the
fixture to exercise it.
Record that the generated client is async only and that interceptors are not
emitted, both because grpc-swift types them on the internal Fory wrapper, and
describe the per-thread marshalling.
Name the wire wrapper per service so it is reachable, then drive it from 2000
parallel threads under ThreadSanitizer, asserting no data race and that the
per-thread Fory stays wire-compatible with the module's shared instance. Against
a shared instance TSan flags a race in the type resolver.
The ThreadSanitizer build adds about three minutes and is environment sensitive,
so keep it opt-in for a sanitizer or nightly job while the functional fixtures
still run on every swift-capable run.
An rpc whose Swift name is handle, serviceName, channel, or defaultCallOptions
would clash with a member the generated provider or client inherits, so fail
codegen with a clear message. Cover the reserved names and nested plus imported
request and response payloads.
# Conflicts:
#	compiler/fory_compiler/tests/test_service_codegen.py
#	docs/compiler/compiler-guide.md
#	integration_tests/grpc_tests/generate_grpc.py
@yash-agarwa-l yash-agarwa-l changed the title Grpc swift feat(swift): Add grpc support for Swift Jun 21, 2026
@chaokunyang

Copy link
Copy Markdown
Collaborator

@yash-agarwa-l Please git merge apache/main first to address the conflicts

@yash-agarwa-l

Copy link
Copy Markdown
Contributor Author

I already did that, so I'm not sure why it's still showing up. Let me mark it as ready for review and double-check.

@yash-agarwa-l
yash-agarwa-l marked this pull request as ready for review June 21, 2026 14:06
The SwiftPM build-and-run round-trip and the ThreadSanitizer marshaller test
need the Swift toolchain, so move them out of the compiler pytest suite (where
they only skipped) into a SwiftPM package under integration_tests/grpc_tests/swift.
The common-root package limitation stays pinned as a build-free generation check.
@yash-agarwa-l yash-agarwa-l changed the title feat(swift): Add grpc support for Swift feat(compiler): Add grpc support for Swift Jun 21, 2026
Mark the generated message wrapper @unchecked Sendable, a transient single-owner
carrier that is serialized synchronously and never shared, so grpc-swift's async
APIs accept it under Swift 6 strict concurrency, and make its value immutable.
Wrap the service descriptor method list to stay within the line limit. Document
that companions compile in Swift 5 language mode until generated models are
Sendable.
@yash-agarwa-l
yash-agarwa-l marked this pull request as draft June 21, 2026 20:16
Point the shared generator at the Swift package's generated sources and run the
relocated marshaller round-trip and concurrency tests with `swift test` when the
toolchain is present, so they execute outside the JVM-driven suite
@yash-agarwa-l

Copy link
Copy Markdown
Contributor Author

Hi @chaokunyang, small update and one thing I'd like your call on.

The Swift gRPC codegen is done and Swift↔Swift round-trips fine, but Java↔Swift interop fails. After digging in, it looks like a gap in the Fory Swift core serializer, not the gRPC code.

When refTracking is on, Swift writes NOT_NULL_VALUE (0xff) for a struct message, while Java/Python/Rust write REF_VALUE (0x00) and reserve a ref slot. (Go reaches REF_VALUE only because grpc-go marshals a pointer; its value-struct path opts out like Swift.) Without the reserved slot, the ref-id counters drift and Java's decode hits Index -1. I checked it's just that one flag byte by flipping it. The reason is that Swift structs are isRefType = false, so foryWrite skips tracking even when it's on. This might be on purpose, since structs are value types. Rust handles the same case by writing REF_VALUE + reserve_ref_id() for value-type structs under tracking.

If that sounds right to you, I'd like to do the same for Swift. One thing to flag: it changes tracking-mode output for all Swift structs, not just gRPC. Tracking is opt-in and off by default, so the impact should be small, but it's a core wire-format change, so I wanted to ask first. No gRPC module changes; the contract stays refTracking:true.

Two questions:

  1. Would it be fine to make Swift value-type structs take part in ref tracking?
  2. Is the current opt-out on purpose, or just not done yet? Want to be sure I'm not missing a reason it writes NOT_NULL_VALUE.

Since it's a wire-format change, I'm happy to do whatever works best for you here, a separate issue or PR, or just keep it on this PR. Let me know.

@yash-agarwa-l
yash-agarwa-l marked this pull request as ready for review June 30, 2026 18:01
@chaokunyang

chaokunyang commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

When ref tracking is enabled, java/python will still push an id -1 to ref reader, and when inner struct invoke RefReader.reference(), it will skip reference for -1 . Could you check this logic? NOT_NULL_VALUE should not be an issue. Use REF_ID is just working around the root cause.

  /** Stores {@code object} under an already reserved read ref id. */
  @Override
  public void setReadRef(int id, Object object) {
    if (id >= 0) {
      readObjects.set(id, object);
    }
  }

There are some code invoking preserveRefId(-1), it's exactly for such cases:

      if (refMode != RefMode.NULL_ONLY || buffer.readByte() != Fory.NULL_FLAG) {
        refReader.preserveRefId(-1);
        return readContext.readNonRef(fieldInfo.typeInfo);
      }

return symbols

def _swift_grpc_method_name(self, method: RpcMethod) -> str:
return self.safe_member_name(method.name)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reject or rename an underscore-only RPC method

FDL accepts an RPC named _, but the generated member remains _, producing declarations and references such as static let _, func _, and Methods._ that Swift cannot use as identifiers. Please generate a referencable name or reject this RPC name during preflight.

"// loop, so the payload must be Sendable. The carrier itself only stores that",
"// payload, so @unchecked covers the wrapper while Value carries the guarantee.",
(
f"struct {base}Message<Value: Serializer & Sendable>: GRPCPayload,"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Align the Sendable constraint with generated model types

This constraint applies to every request and response wrapper, but generated public structs, enums, unions, and classes do not conform to Sendable. Swift 6 therefore rejects even unary providers and clients when Message<Request/Response> is instantiated, not only client-streaming and bidi as the guide states. Please add a safe Sendable strategy plus Swift 6 build coverage, or state that the entire companion currently requires Swift 5 mode.

Comment thread docs/grpc/swift.md Outdated
```swift
// Package.swift
dependencies: [
.package(url: "https://github.com/apache/fory.git", from: "1.2.0"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Require a Fory version that contains this runtime support

from: "1.2.0" allows versions whose Swift Serializer lacks Target, and even 1.6.1 lacks this PR's tracked-value root ref-slot fix required for Java/Swift interoperability. Please use the first release containing both the compiler and runtime changes, or follow the existing $version convention.

final class RoundTripTests: XCTestCase {
func testInProcessAllStreamingModes() async throws {
let group = MultiThreadedEventLoopGroup(numberOfThreads: 1)
defer { try? group.syncShutdownGracefully() }

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use async cleanup in the async test

The current Java/Swift CI job reports async-context warnings for this blocking shutdown, lines 118/124, and main.swift:456; Swift 6 upgrades at least one to an error. Please use awaited get() or structured async lifecycle cleanup while preserving failure-path cleanup. Swift changes are required to compile without warnings.

Comment thread .github/workflows/ci.yml
- name: Run Swift gRPC package tests
run: |
cd integration_tests/grpc_tests/swift/interop
swift test

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Run the concurrency regression under ThreadSanitizer

This runs only plain swift test. The sole --sanitize=thread command is behind FORY_SWIFT_TSAN=1 in another script that this workflow never invokes, so the test and PR description's TSan safety claim is not exercised by CI. Please add a sanitized invocation, preferably filtered to the marshaller concurrency test.

fi
# Swift toolchain tests (generated marshaller round-trip and concurrency). These
# need the Swift toolchain rather than the JVM, so they run in their own package.
if command -v swift >/dev/null 2>&1 && [ -d "${SCRIPT_DIR}/swift/interop" ]; then

@chaokunyang chaokunyang Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honor the selected test classes for Swift work

This block ignores TEST_CLASSES, so ./run_tests.sh GoGrpcTest still resolves and runs the Swift package whenever Swift is installed. Please put both the Swift package tests and the SwiftGrpcTest build behind has_test_class "SwiftGrpcTest", while retaining the tool-availability check before invoking Swift.

fory-compiler declares requires-python >=3.8, but cli.py and the Swift gRPC
service generator used PEP 585 builtin generics and PEP 604 unions in function
signatures. Those evaluate at import time, so foryc raised TypeError on startup
under 3.8 and 3.9. Use typing.List/Dict/Set/Tuple/Optional, matching the other
ten service generators.
The module used PEP 585 builtin generics in annotations, which Python 3.8
evaluates at import time. Match the compatible syntax now used by cli.py and
the Swift gRPC service generator.
The FDL, proto, and FlatBuffers parsers annotated a return type with a PEP 585
builtin generic. Importing the CLI loads all three eagerly, so foryc raised
TypeError on 3.8 even after the CLI itself was fixed.
fory-compiler declares requires-python >=3.8, but the compiler job runs 3.11, so
annotation syntax newer than the declared floor reached main unnoticed. Import
the CLI on the oldest supported interpreters, which covers every module it
loads eagerly.
The thread-local key held only the generated textual type path, so every
default-package schema produced org.apache.fory.grpc.ForyModule. Two generated
targets in separate Swift modules shared that key, and the second wrapper reused
the first module's Fory without its own registrations. Build the key from
String(reflecting:), which is module-qualified at runtime.
Enum-style packages nest their types under a namespace enum, but that enum is
itself a file-scope declaration. Preflight recorded no symbol for it, so two
schemas sharing a top-level package component passed and Swift then rejected the
duplicate declaration. Record the namespace owner so generation fails first.
The symbol collector returned a set, so a name declared twice by one schema
collapsed to a single entry and preflight reported no collision. Report every
declaration with the Swift scope that contains it, keeping duplicates, so the
generated helper against a same-named message, flattened helper names, and names
that normalize to the same identifier all fail before generation.
XCTAssertEqual(fromMarshaller, probe)
}

func testSeparateModulesKeepOwnRuntimeOnOneThread() throws {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make the runtime-key regression test reproduce the old collision

GrpcFdl and GrpcFbs are both compiled into the single ForyGrpcGenerated Swift target, and the old textual keys were already different (GrpcFdl.ForyModule versus GrpcFbs.ForyModule). This test therefore passes even if the String(reflecting:) fix is removed. Please exercise two Swift targets/modules that expose the same textual helper path, ideally default-package ForyModule, so the test fails on the old implementation.

Comment thread docs/grpc/swift.md
hooks would expose that wrapper. Use a custom channel or server configuration for
cross-cutting concerns instead.

Swift models put each package under a nested `enum` namespace, so two schemas that

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the namespace limitation for the new preflight

The compiler now rejects schemas with the same root namespace before writing either file, so this section no longer describes the actual failure point. The separate-module workaround also requires separate generation invocations because one invocation runs the shared collision preflight before module assignment is known. Please document the preflight failure and that requirement.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The preflight wording is closer, but the separate-invocation workaround does not work for the service-import example above. validate_swift_files() recursively collects imports, so compiling demo.greeter still includes demo.shared in the graph and rejects the duplicate Demo even if the shared schema was generated in another invocation. Separate invocations only help unrelated schemas; an import graph currently needs disjoint top-level packages. Please narrow the workaround accordingly.

… services

The wrapper is emitted only when a service declares methods, but preflight
registered it unconditionally, so an empty service rejected a schema type of the
same name that never collided. Gate the symbol on the service having methods and
assert declared symbols match the emitted ones.
The namespace section still described an invalid redeclaration at build time,
but the compiler now rejects a shared root namespace before writing either file,
and the separate-module workaround needs one invocation per module because a
single invocation preflights every schema together. The language mode section
also predated the Sendable payload constraint, which applies to all call
shapes rather than only the streaming client.
Both marshaller regression tests asserted coverage they did not have.

testSeparateModulesKeepOwnRuntimeOnOneThread claimed to guard the module-scoped
ForyRuntime key from d2a47f2, but it used GrpcFdl and GrpcFbs, which compile
into one target and carry packages, so their keys were already distinct under the
old scheme and it stayed green when that scheme was restored. Add two
package-less schemas generated into their own SwiftPM targets: both emit a bare
ForyModule, so their generated key expressions are identical and only runtime
module qualification separates them. Each needs its own foryc invocation, since
one invocation preflights its schemas together and rejects the duplicate
declaration. Against the old "org.apache.fory.grpc.{module}" scheme the new test
fails with "Type not registered: DefaultPackageTwoRequest is not registered".
The old test is renamed to testPackagedSchemasKeepOwnRuntimeOnOneThread, which
is what it actually covers.

The concurrency test claimed to run under ThreadSanitizer in CI, which ran only
plain swift test. The one --sanitize=thread command sat behind FORY_SWIFT_TSAN
in run_tests.sh, which the workflow never invokes and nothing sets. That command
was also broken on macOS rather than merely unreachable: SwiftPM launches XCTest
bundles through the platform-signed swiftpm-xctest-helper and xctest, and dyld
refuses to load the sanitizer runtime into them, while SIP strips
DYLD_INSERT_LIBRARIES. A swift-testing runner is a plain executable and has no
such restriction, so the suite moves to swift-testing and CI runs it with
--sanitize=thread --disable-xctest. Reverting the per-thread Fory to a shared
instance makes TSan report 70 races in TypeResolver.finalizeTypeMeta and exit 1,
so a race fails the step without TSAN_OPTIONS. The step greps for the suite
result so it cannot decay into a zero-test no-op.

Delete FORY_SWIFT_TSAN and its run_tests.sh block: nothing set it, its command
did not work, and CI now owns the sanitized run.
…eholder

from: "1.2.0" advertised a floor that permits releases whose Swift Serializer
lacks Target, and that predate the tracked-value root ref-slot fix needed for
Java/Swift interop.

Naming the first release containing both is not possible yet: df15e7c is in no
tag and is not an ancestor of v1.7.0-rc1, so no cut release carries it and any
concrete number would be a guess. Use the existing convention instead and match
docs/object-serialization/swift/index.md verbatim with exact: "$version".

ci/release.py leaves the placeholder alone because VERSION_PATTERN matches only
numeric versions, and the adjacent grpc-swift line does not match
_is_release_doc_line, so a Fory release cannot rewrite grpc-swift's version.
The in-process round-trip test and the interop client shut down their gRPC and
NIO resources from `defer` blocks using syncShutdownGracefully() and
EventLoopFuture.wait(). Both are unavailable from asynchronous contexts, so the
Java/Swift CI job reported four warnings and the Swift 6 language mode turns the
syncShutdownGracefully() one into an error. wait() on an event loop thread can
also deadlock, which is why NIO marks it unavailable rather than merely noisy.

`defer` cannot await, so the test collects an async teardown step per resource
and unwinds them in reverse on both exit paths, and the client closes its channel
explicitly on the success and failure paths. Teardown behaviour is unchanged:
verified by instrumenting each step and observing channel, server, then group run
when the body succeeds and when it throws.

Our sources now build with zero warnings; the remaining ones come from
third-party checkouts.
Bring the Swift gRPC branch up to date with main, which has moved 28 commits
ahead including the 1.6.1 release version bump.
…workaround

The Swift package tests were gated only on `command -v swift`, so
./run_tests.sh GoGrpcTest resolved and ran them whenever a toolchain was
installed, and the SwiftGrpcTest release build sat in a second block. Both now
share one has_test_class "SwiftGrpcTest" guard, keeping the tool-availability
check before Swift is invoked.

The namespace section also told users that generating into separate Swift modules
with one foryc invocation each works around a shared root namespace. That only
holds for unrelated schemas. validate_swift_files builds its graph through
collect_schema_graph, which walks imports recursively, so compiling demo.greeter
still includes demo.shared and rejects the duplicate Demo even when the shared
schema was generated in another invocation. An import graph needs disjoint
top-level packages, and the section now says so.
safe_member_name routes the rpc name through to_camel_case, which strips
underscores, so an all-underscore name normalizes to empty and safe_identifier
returns its "_" fallback. rpc _, __, and ___ all reached it and emitted func _,
static let _, and Methods._. swiftc rejects func _ outright, while static let _
parses as a wildcard binding and fails only where Methods._ is referenced.

Reject instead of renaming: an invented name would be arbitrary and could collide
on its own. The check joins the existing method-name preflight, which the CLI
reaches before writing any file, so it exits with a clear error and leaves no
partial output. That preflight now covers unusable names, reserved members, and
duplicates, so it is named _check_swift_grpc_method_names.

Document both rpc-name rules, including the reserved-member list that was
already enforced but never written down.
swift_declared_symbols() iterated only the schema's top-level enums, unions, and
messages, so nested declarations never reached the collision preflight. A schema
such as message Parent { message my_type {} message MyType {} } passed and then
emitted two public struct MyType inside Parent. The failure is semantic rather
than syntactic, so swiftc reports invalid redeclaration only at type-check.

Walk nested messages, enums, and unions recursively and key each declaration by
its parent Swift scope. Enums and unions stop the walk because only Message
carries nested collections. Nested names keep the bare normalized form, so the
flatten namespace prefix still applies to the top-level owner alone.

test_swift_nested_normalized_name_collision_fails_preflight described package
namespace nesting rather than an IDL nested type, so it is renamed to
..._packaged_..., and real nested cases now have their own tests, including a
deeper level and same-named types under distinct parents.
…d code

The generated models declare Equatable but not Sendable, and the gRPC wire
wrapper constrains its payload to Sendable, so Swift 6 strict concurrency
rejects every call shape rather than only the streaming ones.

Say so in the model documentation as well, because the limit applies to any
Swift 6 code that moves a generated value across an isolation boundary, not only
to gRPC users. Drop the claim that Swift 6 support follows once models are
Sendable: messages using ref or weak fields generate classes with mutable stored
properties, which cannot declare Sendable at all.
@chaokunyang
chaokunyang merged commit 676253b into apache:main Aug 24, 2026
71 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants